home *** CD-ROM | disk | FTP | other *** search
/ WINMX Assorted Textfiles / Ebooks.tar / Text - Tech - Hardware - Hardware Book, The (TXT, HTML).zip / robots.txt < prev   
Text File  |  2001-06-07  |  230b  |  8 lines
  1. # robots.txt
  2. # This file specifies what harvesting robots are allowed to index and not.
  3. #
  4. # For information on the format: 
  5. # http://www.robotstxt.org/
  6. #
  7. # Everything not explicitly disallowed is considered fair game to retrieve.
  8.